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In this article, we have explored the issues that Ministries of Education confront in 
their large-scale assessment policies and practices as literacy curricula expand to 
include the new literacies of information and communication technologies. Based on a 
series of interviews with Ministry personnel, we have described their current progress 
to include the new literacies, plans for the future, and the dilemmas and challenges in 
considering the demands of new times. We argue for a more balanced approach to 
assessment and accountability, one that recognizes the limitations of current public 
accountability measures and situates them within a broader assessment framework. 
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Dans cet article, les auteures etudient les enjeux auxquels sont confrontes les 
ministeres de I'Education dans leurs politiques et methodes en matiere d'epreuves 
communes au fil de I'integration des technologies de I'information et de la 
communication au sein des programmes de literatie. A partir d'une serie d'entrevues 
realisees aupres de fonctionnaires de ces ministeres, les auteures font le point sur 
I'inclusion des nouvelles litteraties, la planification du developpement envisage, les 
dilemmes et les defis qu'impliquent les exigences actuelles. Elies preconisent une 
approche plus equilibree de revaluation et de I'imputabilite, laquelle tiendrait compte 
des limites des mecanismes d'imputabilite actuels et les situerait dans un cadre 
d' evaluation plus vaste. 

Mots cles : tests communs, methodes d' evaluation canadiennes, politiques, nouvelles 
litteraties. 


Government and industry around the world are challenging education 
systems to prepare successful learners, workers, and citizens for fhe new 
knowledge-based economy. One crifical componenf of fhis economy. 
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competence in new literacies, includes "how to read and produce the 
kinds of texts typical of fhe emerging informafion and multimedia age" 
(Semali & Fueyo, 2001, n.p.). 

The challenge lies in debafe around whaf forms of liferacy fo feach 
and whaf pedagogical options are mosf appropriafe for feaching 
differenf forms of liferacy. Educafors, who are af a crifical juncfure, are 
exhorfed fo expand fhe nofion of liferacy nof only by govemmenf and 
indusfry, buf also by professional associations such as The Nafional 
Council of Teachers of English and fhe Infemafional Reading Associafion 
(1996). Yef, as Eloffman, Paris, Salas, Patterson, and Assaf (2003) nofe, 
basic skills-fesfing programs, which are increasing in frequency, exerf a 
powerful influence on insfrucfion af all levels, especially in fhe U.S.A. 
and U.K. In Canada, sfandards-based provincial assessmenfs are faking 
on new significance wifh fhe publication of "reporf cards" fhaf compare 
and rank schools and school disfricfs (Fraser Insfifufe, 2005). 

In fhis article, we have explored liferacy accounfabilify and 
assessmenf in fhe Canadian confexf. Our examination of minisfry of 
educafion websifes showed fhaf resulfs of province-wide assessmenfs 
are multi-purpose; fhey defermine individual sfudenf liferacy 
affainmenfs, inform parenfs of fheir children's liferacy proficiency, 
monitor insfrucfion of fhe curriculum, and hold schools and school 
boards accounfable. Thus, fhese assessmenfs have increasing imporf; yef 
minimum analysis has occurred of fhe relevance of fradifional prinf- 
based liferacy, promoted in province-wide assessmenfs, fo fhe new 
economic and social realities. In fhe U.S. confexf. Leu, Afaya, and Coiro 
(2002) found fhaf "fo a remarkable exfenf, sfafe assessmenfs in liferacy 
complefely exclude any of fhe new liferacies fhaf will define our 
sfudenfs' fufure, nor do fhey have plans fo do so" (p. 8). As Canadian 
liferacy educafors and researchers, we felf fhaf if was crifical fo examine 
fhe currenf sifuafion in Canada, particularly because we acknowledge, in 
principle, fhe view of Johnson and Kress (2003) fhaf "despite years of 
debafe on fhe nafure of liferacy and curriculum and ensuing policy 
directions, if is assessmenf— if s weighting in fhe polifical culfure and fhe 
means of enforcing fhaf culfure, which will guide whaf is faughf and 
how if is faughf" (p. 11). 
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LITERACY FOR A WORLD IN TRANSITION 

We are living in the midst of a period of dramafic global economic 
change, co-evolving wifh and fuelled by an equally dramafic 
fechnological revolufion. These forces combine fo radically alfer our 
work, civic, and personal lives (Casfells, 1996; Drucker, 1993; Gee, Hull & 
Lankshear, 1996). In our work lives, fhe shiff fo a knowledge-based 
economy is infrinsically linked wifh fhe quickening speed of knowledge 
change (and obsolescence), rising sfandards of qualificafions for 
"knowledge workers," and increased infegrafion of ICT [Informafion and 
Communicafion Technologies] in fhe workplace (Organizafion for 
Economic Co-operafion and Developmenf, 2001, p. 3). The speed of 
knowledge change is increasing so rapidly fhaf no matter how currenf 
knowledge learned in schools, if will likely soon be obsolefe. Moreover, 
fhe new economy demands nof only new types of knowledge, buf also 
new ways of fhinking and faking action, and new learner skills and 
characferisfics (Casfells, 1996; Gee, Hull & Lankshear, 1996). These 
include fhe abilities fo work creatively and productively in feams, 
frequenfly in nefworked environmenfs; fo engage in fechnical and 
sysfems fhinking; fo manipulafe absfracf concepfs and symbols; and fo 
learn, in facf, how fo learn (New London Group, 1996). Lifer acy 
capacities fhaf govemmenf and business deem mosf relevanf for fhe new 
economy are "fhe abilifies fo 'read' a range of prinfed, elecfronic and 
visual fexfs; masfer fhe new communicafion fechnologies via spoken and 
wriffen language; locale, manage, evaluafe and use informafion or 
knowledge; and engage critically wifh media and ofher texfs" (Lonsdale 
& McCurry, 2004, p. 32). 

Arguably, in increasingly profound ways, fhe changes ouflined 
above define nof only learning, buf also reading, wrifing, and whaf if 
means fo be lifer afe (Leu, Kinzer, Coiro & Cammack, 2004). These 
changes prompf reconcepfualized notions of liferacy fhaf encompass 
new media and ICT (Kinzer & Leander, 2003; Kress, 2003; Lonsdale & 
McCurry, 2004). These new forms of liferacy are bofh verbal and multi- 
modal. For example, a web page selling goods or services will likely 
combine fexf, music, sound effecfs, and visual images. Or, an 
adverfisemenf for a movie, radio sfafion, or DVD will offer animation 
and full motion video. Web pages, websifes, e-mail, chaf, insfanf 




Accountability, Assessment, and the Literacies o f Information 


805 


messaging, video, games, anime, listserves, group lists, hypertext, 
hypermedia— elementary and high-school students navigate these new 
genres in their daily lives, forms that are dynamic, interactive, and 
evolving in sophistication, that require a new set of compefencies for 
parficipafion in fhe new fimes and world orders. 

These new liferacies are nof old or fradifional prinf liferacy 
fransf erred fo on-line environmenfs; fhey involve unique processes and 
skills (Coiro, 2003; Kymes, 2005; Leu, 2002; Sufherland-Smifh, 2002). Leu, 
Afaya, and Coiro (2002) acknowledge fhaf 

while the precise definition of 'new literacies' will never be complete . . . they 
include a) rapidly locating the most useful information within complex ICT 
networks such as the Internet; b) reading and critically evaluating that 
information for validity and utility; c) writing effectively with word processing 
software; and d) communicating information clearly to others with e-mail. (p. 1) 

In a recent Information and Communication Technologies in Schools 
Survey in Canada, Plante and Beattie (2004) found fhaf nearly all schools 
are cormecfed fo fhe Infernef, wifh an average sfudenf-fo-compufer rafio 
of 5:1 and an average of 72 compufers per school. More imporfanf, as 
Lenhart, Madden, and Hiflin (2005) defermined, fhe new liferacies of ICT 
are rapidly becoming everyday liferacies as feens spend more time 
online. Given fhis sifuafion, assessing sfudenfs' abilifies in ICT liferacies 
is increasingly urgenf and "as fechnology becomes more cenfral fo 
schooling, assessing sfudenfs in a medium differenf from fhe one in 
which fhey fypically learn will become increasingly unfenable" (Bermeff, 

2002, p. 2). 

METHODOLOGY 

We became inferesfed in Ganada's practices wifh new liferacies in large- 
scale assessments after learning of Leu, Afaya, and Goiro's (2002) 
invesfigafion in the U.S.A. in which they interviewed persons in each 
state responsible for literacy assessment about the inclusion of IGT 
literacy on their mandated tests. We decided to replicate this study in 
Ganada. Table 1 provides an overview of liferacy/language arfs 
assessmenf af fhe fime of dafa collection for each province. Of fhe 
fhirfeen provinces and territories, two do not use large-scale tests, two 
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chose not to participate in our study, and two did not reply to our 
request to participate. 


Table 1 

Overview of Provincial Language Arts Assessments: Focus, Grades Administered, 
Administration Cycle, and Language Arts Curriculum Publication Date 



Language Arts 

Grades when 

Administration 


Curriculum 



Assessment 

Administered 

Cycle 

Document Dates 



Focus 





British 

• 

Reading 

Grades 4, 7 

Annually for the 

1996 

Columbia 

• 

Writing 

and 10 

3 grades 



Alberta** 

• 

Reading 

Grades 3, 6 

Annually for the 

2000 


• 

• 

• 

Writing 
Viewing 
Word 
Processing 
(grades 6 & 

9) 

and 9 

3 grades 



Saskatchewan 

• 

Reading 

Grades 5, 8 

Bi-annually for 

• 

Elementary 


• 

Writing 

and 11 

the 3 grades 

• 

2002 

Middle years 
1997 






• 

Grades 10, 

11, 121999 

Manitoba 

• 

Reading 

Grade 3 

Annually for the 

• 

K-41996 


• 

Language 

(Teacher 

3 grades 

• 

Middle years 



Arts 

administers) * 



1996 


• 

English 

Grade 6 


• 

Senior years 



Language 

Optional 



1996 



Arts 

Senior 4 
required 




Ontario 

• 

Reading/ 

Grades 3 and 

Annually for the 

• 

Elementary 


• 

Writing 

6 

3 grades 


1-8 1997 



Literacy 

Grade 10 


• 

Grade 9 & 10 

1999 






• 

Grade 11 & 

12 English 
2000 

Quebec 

• 

Reading 

6*'’ year 

Annually for the 

2003 


• 

Writing 

Primary 

5*'’ year 

2 grades 
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Language Arts 

Grades when 

Administration 

Curriculum 



Assessment 

Administered 

Cycle 

Document Dates 



Focus 







Secondary 



New Brunswick 

• 

Reading 

Grades 2 and 

Annually for the 

1998*=^=^* 


• 

Writing 

7 

2 grades 


Prince Edward 

• 

Reading 

Grade 9 (15 

Annually 

1998*=^=^* 

Island’^** 

• 

Visual 

literacy 

year-olds) 

(OECD/PISA) 


Nova Scotia 

• 

Narrative/ 

Grades 6, 9 

Annually for the 

1998*=<-=^* 



Information 

Reading 

and 12 

3 grades 



• 

Writing 

tasks 





• 

Visual/medi 

a text 




Nfld/ 

• 

Process 

Grades 3, 6 

Annually for the 

1998*=^*=<- 

Labrador 

• 

Writing 

Oracy 

and 9 

3 grades 



• 

Narrative/ 

Information 

Reading 





• 

Viewing 





• 

Electronic 

Text 




Yukon 

• 

Reading 

Grades 3, 6 

Annually for the 

1 (^(^^***** 


• 

Writing 

and 9 

3 grades 



• 

Viewing 




Northwest 

• 

Reading 

Grades 3, 6 

Annually for the 

1998 

Territories*'*' 

• 

Writing 

and 9 

3 grades 



• 

Viewing 





• 

Word 
Processing 
(grades 6 & 

9) 




Nunavut 

N/A 

N/A 

N/A 

2000 (uses 

(no large scale 





Alberta Learning 

assessment 





curriculum) 

program in 
place) 







* Department provides assessment materials and professional learning 
opportunities for teachers in support of assessment of critical competencies in 
reading for students in the beginning of grade 3. 
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** Alberta includes the Northwest Territories in its provincial large-scale 
assessment 

Prince Edward Island has no provincial large-scale assessment program but 
participates in the international large-scale assessment program OECD/PISA 
(Organization for Economic Co-operation and Development/Program for 
International Student Assessment). 

**** Implements the Atlantic Canada English Language Arts Curriculum 
***** Yukon Department of Education, Public School Branch relies on the 
curriculum produced by other jurisdictions, especially the British Columbia 
Ministry of Education. 

Between September 2003 and April 2004, we conducted telephone 
interviews with one person in each of seven provinces. All interviewees 
were people at senior levels in their assessment and evaluation 
departments. Ministries of Education fhaf parficipafed in fhis sfudy 
represenf 83 per cenf of fhe Canadian public school population. We 
analyzed inferview dafa quanfifafively by adapting Leu, Afaya, and 
Coiro's (2002) scoring rubric. Unlike fhe U.S. sfudy, we exfended fhe 
quanfifafive dafa by examining inferviews for fhemes. 

We asked Canadian parficipanfs fhe same quesfion fhaf Leu, Afaya, 
and Coiro (2002) used in fhe American sfudy: To whaf exfenf have sfafes 
in fhe U.S. [i.e., provinces in Canada] included in fheir sfafe assessmenf 
pracfices fhe assessmenf of word processing skills, reading and critical 
evaluation of informafion on web pages, fhe use of e-mail, or any of fhe 
ofher new liferacies of ICT? In addition fo fhese areas fhaf were covered 
in fhe U.S. sfudy, we also asked abouf inclusion of visual liferacy. 

Inferviews focused on five areas: reading, wrifing, sfudenfs wifh 
special needs, plans for fhe fufure, and curricular area roles and 
responsibilities. The firsf parf of the interview focused on reading and 
asked abouf fhe assessmenf of abilifies fo locafe, evaluafe, and 
comprehend informafion on fhe Infemef, and inferpref informafion in 
visual forms. To learn abouf wrifing assessmenf, in fhe second parf of fhe 
interview, we asked about the option to use a word processor and the 
ability to express understanding in visual forms. Lor bofh fhe reading 
and wrifing areas, we asked respondenfs reasons for including or nof 
including fhe above in fheir presenf provincial assessmenfs, and whefher 
fhey foresaw any change ahead in including fhese areas. The fhird parf 
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of the interview addressed diversity by asking whether special education 
students could use a word processor for the provincial writing 
assessment and what special conditions were available for students for 
whom standard forms of reading and writing were not an option. The 
fourth part of the interview asked about timelines for implementing the 
following: a) the use of word processors during the provincial writing 
assessment; and b) evaluating students' ability to use e-mail, 
comprehend online materials, and locate and evaluate online materials. 
The fifth segment of the interview questioned whether each of the 
following areas is regarded as a reading assessment issue, a writing 
assessment issue, a technology assessment issue, or a non-issue for 
assessment: using word processors, using e-mail, comprehending online 
materials, and locating and evaluating online materials. The interview 
closed by asking respondents what they foresaw as general future 
directions for their provincial assessments, and what roles present 
assessment measures play in assessment practices and policies. 

The focus of our interview questions represented a limited view of 
ICT literacies; nevertheless, these questions have allowed us to obtain an 
initial understanding of the place of ICT literacies in large-scale 
assessment in Canada. Although ICT literacies may be part of other 
curricular areas, following Leu, Ataya, and Coiro (2002), we limited our 
investigation to literacy curricula. 

RESULTS 

Through analyses of the quantitative data and the interviews, several 
themes emerged that we report with the following headings: ICT literacy 
in present assessments; plans for including ICT literacies in future 
assessments; and the challenges and dilemmas of incorporating ICT 
literacies into large-scale assessment. i 

ICT Literacies in Present Provincial Literacy Assessments 

In Table 2, we have summarized the types of ICT literacies presently 
included on provincial large-scale literacy assessments, aspects of ICT 
literacies that provinces are contemplating including in future literacy 
assessments, and approximate timelines for implementing these changes. 
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Province New Literacies 

Presently Included 

# 1 • Comprehension 

and evaluation 
of online 
information 

• Visual literacy 

• Media literacy 


#2 

#3 

#4 


#5 

#6 


# 7 
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Table 2 

y Included and ICT Literacies Contemplated for 
Reading and Writing Assessments 


New Literacies Approximate 

Contemplated for Time Table for 


Inclusion 


Change 


• Use of word • 2-4 years 

processor 

• E-mail 

• Locating on-line 
information 

• Locate, 
comprehend 
and evaluate in 
an on-line 
environment 

Comprehension of • 5-10 years 

on-line 
materials 

• Locate, • 3-5 years 

comprehend 

and evaluate on- 
line materials 

• Word • 2-4 years 

processing 

• Locate, 
comprehend 
and evaluate on- 
line materials 

• Word • 5-10 years 

processing 

• Word • Unknown 

processing 

• Locate, 
comprehend 
and evaluate on- 
line materials 

• Locate, • Unknown 
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Province New Literacies New Literacies Approximate 

Presently Included Contemplated for Time Table for 

Inclusion Change 

comprehend 
and evaluate on- 
line materials 


Reading. One provincial reading assessment included questions at 
the grades-6 and -9 levels about comprehension and evaluation of 
information on the Internet. These were presented as paper and pencil, 
not on-line, tasks. For example, in the grade-6 reading assessment, 
students identified fhe disfincfive feafures of web pages and how fhis 
fexf form differs from fradifional prinf or book forms. These abilifies 
were also represenfed in fhe assessmenf programs in fwo ofher 
provinces, buf in subjecf areas ofher fhan reading. In one of fhe 
province's wrifing assessmenf, secondary sfudenfs were given a fopic fo 
wrife on and received a reading file abouf fhis fopic, which included 
documenfs from fhe Infernef. Sfudenfs complefed fheir reading file by 
obfaining informafion from pre-selecfed Infernef sifes or by 
independenfly searching for informafion on fhe Infernef. However, 
sfudenfs were nof evaluafed on eifher fheir comprehension or evaluation 
of fhese Infernef resources buf rafher on fheir final written producf. 
Anofher province assessed "sfudenf skill in accessing, processing, and 
communicating informafion in a previous year," buf in fhe fechnology, 
nof liferacy, assessmenf. 

The same province fhaf included comprehension and evaluafion of 
on-line informafion in fheir grade-6 provincial reading assessmenf also 
assessed sfudenfs' abilify fo comprehend ideas in formafs ofher fhan 
wriffen fexf. The questions abouf a web page described above, for 
example, required visual liferacy. And, using an adverfisemenf fhaf used 
images of a sfuffed animal and a young girl fo sell a producf, sfudenfs 
were asked why fhe arfisf included fhose characfers, who fhe fargef 
audience mighf be, and why everyfhing excepf fhe producf was 
porfrayed in black and while. Anofher ifem was an illusfrated 
informafion fexf wifh comprehension quesfions abouf fhe illusfrafions. In 
fhree ofher provinces, comprehension of visual fexfs was parf of ofher 
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curricular area assessments such as social studies or math, but not 
reading: "The visual representations are used as aids but they are not 
being targeted for assessment. For example, chronological timelines have 
been used in the large-scale tests for fhe primary level and geographical 
maps in fhe large-scale assessmenf for social sfudies" (Province 3). 

Writing. Wifh fhe excepfion of one province, provincial assessors did 
nof allow fhe use of word processors for large-scale wrifing assessmenfs. 
As a pilof projecf, one province offered schools fhe opfion of using a 
word processor for fhe secondary wrifing assessmenf wifh fhe condifion 
fhaf spell checkers be blocked; however, few schools exercised fhe 
opfion. More typically, writing assessments were restricted to paper and 
pencil where "even e-learners or home-schoolers have to come and sit 
and write the exams as pen and paper" (Province 4). 

One province included visual forms of expression in ifs wrifing 
assessmenf (e.g., sfudenfs responded fo a poem by drawing a carfoon 
and used a Venn diagram fo show fheir undersfanding of an informafion 
texf). Respondenfs from fwo provinces explained fhaf visual forms of 
expression were parf of classroom assessmenf because fhey are 
curriculum-based, buf visual forms of composing are nof parf of fhe 
provincial wrifing assessmenf. In one of fhese provinces, assessors 
included visual represenfafion in a large-scale assessmenf in fechnology 
several years prior, nof wrifing, in which sfudenfs were asked fo "fell 
whaf fechnology means fo you by drawing picfures and wrifing abouf if 
in fhe space below" (Province 6). In fwo of fhe provincial wrifing 
assessmenfs, sfudenfs were encouraged fo use graphic organizers (webs 
and diagrams) as parf of prewrifing, buf fhese were nof evaluafed wifh 
fhe final written producf. 

Diverse Students and Large-Scale Reading and Writing Assessments 

In all buf one province, special educafion sfudenfs who had designated 
status and permission were allowed to use word processors for fhe 
provincial wrifing assessmenf. Some respondenfs explained fhaf 
provincial policy provided special educafion sfudenfs in large-scale 
assessmenf sifuafions wifh fhe same learning adapfafions they would 
use in their classrooms. One respondent noted that in these cases, "spell 
and grammar checks must be turned off, and wrifing monifored by fhe 
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teacher. We want students to use the word processor more as a tool for 
their writing, not something that will be an aid" (Province 2). 

All provinces provided a variety of special condifions for sfudenfs 
for whom sfandard assessmenf forms of reading and wrifing were nof an 
opfion, condifions fhaf offen involved fhe use of ICT. They reporfed fhaf 
fhese included whaf is normally done in fhe classroom, such as alfemafe 
settings, English as Second Language dictionaries, addifional time, an 
assisfanf, access fo a scribe, assisfed fechnology (e.g., exfra large fonf), 
exfra lighting, and differenf response formafs (e.g., oral insfead of 
wriffen). 

Plans for Including ICT Literacies in Future Assessments 

From our inferviews, we found fhaf mosf provinces were plarming fo 
permif all sfudenfs fhe choice fo use word processors for fhe wrifing 
assessmenf. Two respondenfs explained fhaf implemenfafion of fhis plan 
was in fhe immediafe fufure and was already being pilofed in one of 
fhese provinces (as menfioned above). Three respondenfs indicafed fhis 
sfep in fheir provinces is five fo fen years in fhe fufure. Two ofher 
respondenfs indicafed an unknown fimeline, explaining fhaf compufers 
had fo be complefely infegrafed into classrooms before considering word 
processing a sfandard practice on fhe provincial assessmenf. 

Our dafa revealed fhaf only one province plans fo evaluafe sfudenfs' 
abilifies fo use e-mail, buf if will be in fhe confexf of pen and paper 
because "word processing is jusf a mechanism for sfudenfs fo do a final 
copy" (Province 1). Three respondenfs sfafed fhaf fhe ability to use e- 
mail could be included in technology assessment programs to 
correspond with new provincial ICT performance sfandards, buf no 
definife plans were in place. In fhree provinces, e-mail wrifing was parf 
of fhe curriculum fhus indicating if was probably included in classroom 
assessmenf, buf fhere were no immediafe plans fo include if in large 
scale assessmenf. 

Five respondenfs reporfed fhaf if was likely fhaf fheir provinces 
would include in fheir fufure reading assessmenfs locafing, evaluating, 
and comprehending on-line maferials. One of fhese provinces was 
progressing quickly in fhis direction because fheir currenf assessmenf 
already included paper and pencil ifems fhaf fargefed fhese abilifies. 
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Another province was "talking about asking students to read a web page 
as a text, but not asking them to use links and other on-line features. 
"We're going to present the page view and ask questions about that 
information" (Province 2). 

Other respondents foresaw longer fimelines in fheir provinces for 
implemenfing fhese changes fo fheir reading assessmenfs. Two provinces 
had no definife plans fo include locafing and evaluafing on-line 
informafion. In one of fhese provinces, fhe lasf Informafion Technology 
assessmenf included locafing informafion buf nof evaluafing or 
comprehending if. One respondenf explained fhaf fhese abilifies could be 
evaluafed fhrough alfemafive mefhods fo fradifional large-scale 
assessmenfs, such as sfudenf files or porffolios, collecfed by schools, 
confaining evidence of fhose abilifies (Province 3). 

In some provinces, fhere was considerable planning for on-line 
assessmenf programs where "sfudenfs wrife fhese exams direcfly on fhe 
Nef. Buf if will sfill be selecfed responses (i.e., multiple choice) and 
reading comprehension" (Province 4). This respondenf nofed fhaf "one 
of fhe questions could be accessing a websife and reading fhis 
informafion and reporfing on fhis informafion." A pilof projecf was 
underway in fhis province, indicafing fhaf implemenfafion may be 
imminenf. 

Wifh ofher provinces, plans fo include ICT liferacies on large-scale 
assessmenfs were vague. "Af some fime in fhe fufure we will likely be 
using compufers for various reasons. Nof jusf fo allow sfudenfs fo 
respond fo various prompfs buf also in ferms of on-line assessmenf ifself. 
If's quife probably a long way off for us and I fhink af leasf five years 
away— maybe more" (Province 5). These provinces, fhe ones wifh less 
definife fimelines, foresaw more conservafive fufure changes in bofh fesf 
confenf and formafs. "Aparf from revisiting some of fhe approaches fhaf 
we're using in our assessmenfs, for fhe foreseeable fufure we're nof 
looking righf now info new formafs and new approaches" (Province 5). 

Challenges and Dilemmas in Including ICT Literacies on Large-Scale 
Assessment 

Our respondenfs arficulafed a number of challenges and dilemmas fhey 
currenfly face. These included changing conceptions of liferacy. 
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connections between the curriculum and assessment, equitable ICT 
access issues, and the differing polifical and educafional uses of 
assessmenf resulfs 

Conceptions of literacy. Mosf Minisfries were engaged in debafes 
abouf whaf counfs as liferacy as represenfed on fheir large-scale 
assessmenfs. There was a conservafive fendency and a resisfance fo 
include new or whaf was perceived as "informal" liferacies. For example, 
one interviewee commented that, "we still like students to be able to 
write in the traditional forms. We find generally fhaf when sfudenfs are 
using word processing, fhey gef closer fo an e-mail sfyle of wrifing or 
fhey sfarf using chaf room language or acronyms and fhings like fhaf. 
We really af fhis poinf are sfill resisfing" (Province 2). Our respondenfs 
explained fhe possible consequences of allowing fhese less formal 
liferacies on fheir provincial assessmenfs as pofenfially defrimenfal fo 
classroom insfrucfion of fradifional prinf-based liferacies. 

Unfortunately as much as we say our assessments should not drive practice they 
often do and so if we were to replace a formal business letter on one of our 
assessments with a write an e-mail to your best friend about a TV show you saw 
last night, then all of a sudden teachers across the province would be doing that. 
(Province 2) 

Responses about perceived curricular responsibility of fhe use of 
word processors, abilify fo use e-mail, comprehension of online 
maferials, and locafion and evaluafion of online maferials provided a 
window info Minisfry concepfions of ICT liferacies (see Table 3). 

All respondenfs excepf one claimed fhaf fheir provinces considered 
fhe use of word processors a fechnology issue. One respondenf 
disfinguished befween classroom assessmenf, in which using a word 
processor would be a wrifing issue, and large-scale assessmenf, in which 
if would be a fechnology issue. Using e-mail was considered a 
fechnology issue in fhree provinces, a reading and wrifing issue in one 
province, a wrifing issue in one province (however, fhis province did nof 
recognize e-mail as a legifimafe form of wrifing, and did nof include if in 
large-scale writing assessments), a reading, writing and technology issue 
in one province, and a non-issue in another province where the product. 
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Table 3 

Perceptions of Curriculum Responsibility ofICT Literacies in Provincial Assessment 

Programs 


Province 

Use Word 

Processors 

Use E-mail 

Comprehend 
Online Materials 

Locate and 

Evaluate 

Online 

Materials 

1 

Technology 

Reading, 

Writing, 

Technology 

Reading 

Reading and 
Technology 

2 

Technology 

Writing 
(however, is 
not 

recognized as 

legitimate 

form) 

Reading 

Technology 

3 

Writing 

Reading & 

Writing- 

interactive 

Reading (in 
curriculum 
now as "variety 
of formats") 

Reading (part 
of critical 
reading in 
curriculum 
now) 

4 

Technology- 

Technology 

Technology 

Technology 


separate from 

Curriculum 

and Reading- 

and Reading- 


ability to 
communicate, 
organize ideas 
etc 

based 

requires higher 
level mental 

processes 

requires higher 
level mental 

processes 

5 

Technology 

Technology 

Technology 

Technology 

6 

Technology in 

Technology 

Technology in 

Technology in 


large scale 

in large scale 

large scale 

large scale 


assessment. 

assessment. 

assessment. 

assessment. 


writing in 

writing in 

reading in 

reading in 


classroom 

classroom 

classroom 

classroom 


assessment 

assessment 

assessment 

assessment 

7 

Technology 

Non-issue- 
product is 
focus of 

assessment 

not process 

Technology 
and Reading 

Technology 
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not the process, was the focus of evaluafion. Comprehension of online 
maferials in large-scale assessmenf programs was regarded as a reading 
issue in fhree provinces, a reading and fechnology issue in fwo 
provinces, and a fechnology issue in fwo provinces (wifh one of fhese 
viewing if as a reading issue in classroom assessmenf). Locafing and 
evaluating on-line maferials was considered a fechnology issue in four 
provinces (again wifh one of fhese viewing if as a reading issue in 
classroom assessmenf), a reading and fechnology issue in fwo provinces, 
and a reading issue in one province. 

Several people mentioned fhe lack of effecf of fechnology on 
assessing liferacy abilify, such as fhe use of a word processor on a 
sfudenfs' writing. 

It really doesn't mean that they are necessarily better writers or better readers 
because they have given it to you on a typed page because at the end of the day 
it's the typed text as opposed to a hand-written text. (Province 1) 

The issue of the relationship between technology and literacy was a 
common concern, as captured in this response. 

We have to separate the ability to write and to communicate from the technical 
issue, which is using word software or other tools. Comprehension of on-line 
information is more than a technical issue because the stmcture of the Internet is 
a living tool and understanding that and using those tools are higher mental 
processes. (Province 4) 

Relationships between curriculum and assessment. Most respondents 
spoke about the assumption of a coherenf relationship befween fhe 
provincial curriculum and assessmenf. Some provinces had broader 
liferacy curricula in fhe language arfs. Wifhin fhis group, only one 
province spoke of fhe consisfency befween fhe mandafes of curriculum 
and large-scale assessmenf, a key characferisfic of successful sfandards- 
based school reform inifiafives (Darling-Hammond, 2004). 

In our curriculum outcomes, our students are expected to be able to read a 
variety of different kinds of texts. And texts are defined more broadly than 
words on paper. In our assessments, we include narratives and we include 




818 


M. Asselin, M. Early, &M. Filipenko 


poems but we include the visuals because it's part of our notion of reading and 
we include the electronic as part of our notion of reading. (Province 1) 

Those from provinces with narrower literacy curricula explained 
that they did not include new literacies on their current assessments 
because the curriculum basis is not there. 

The need to look at technologies, the need to look at the Internet as a source of 
information and so on— we don't have that in our curriculum. And because we 
try to tie our assessments to the curriculum, we can't in all fairness put too great 
an emphasis on that on our assessments. (Province 2) 

Our respondents indicated that their ministries continued to look to 
the current and forthcoming curricula in planning future large-scale 
assessments. For example, one person explained that their province's 
current reading curriculum stated, "The student should be presented 
with texts on various mediums. This will not give us any other choice 
than asking ourselves what we can offer in terms of large-scale 
assessment" (Province 3). This emerging curriculum basis of reading and 
writing with new technologies also means these new literacies must 
become part of classroom assessment. The challenge is making the link 
with the large-scale assessment program. One person explained, "We 
have already integrated anything that concerns ICT within the 
curriculum and within the classroom assessment. The step ahead would 
be to include that in our large-scale assessment" (Province 4). 

Our respondents also recognized the powerful place of technologies 
in students' reading and writing in and out of school. For example, in the 
province that provided students with printouts of Internet resources for 
the writing assessment, our interviewee discussed how 

the students tend to ignore more and more the documentation that we offer to 
them and just establish their own, so we are telling ourselves that we have no 
other choice but to help our students learn to read and write with these 
technologies. (Province 3) 

The guiding principle for future assessments seems to be that as 
curriculum changes to reflect expanded notions of literacy, so too should 
assessments. 
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Access to ICT. All respondents spoke of the complex and practical 
issues related to technology access in expanding large-scale assessments 
to better capture the new ICT literacies in the curriculum. One 
respondent described this situation as "a tension between the opening 
towards the technologies and also the whole aspect concerning an 
assessment tool that would strive to make sure that all students be 
placed in similar conditions of test administration" (Province 3). 
Concerns were expressed generally about equitable access and 
specifically about the quality of computers and software available for the 
assessment, as well as students' regular opportunities to use word 
processors for composing. 

The main obstacle to be removed as we progress is the issue of access, access to 
technologies in our schools, so we must give access to our students. This varies a 
lot throughout [our province] right now. There are schools where the students 
arrive at school with their own laptops, while some other schools are not very 
well equipped. (Province 3) 

Although nearly all schools in Canada now have computers, and 90 
per cent of schools are connected to the Internet, computer access and 
quality of access, even where computer-to-school ratios are reasonably 
high, varies greatly among schools (Plante & Beattie, 2004). As well, 
differences between students' access opportunities and experiences with 
technology vary within schools. Ministries are aiming for a "level 
playing field" between schools and students before they feel they can 
legitimately consider implementing assessment of ICT literacy. 

It would be seen as an advantage for those students who have the capability to 
use computer equipment as opposed to those who can't because there's a certain 
software they might have access to that other students wouldn't have access to. 
(Province 5) 

One respondent reported that the matter of computers in schools 
was a "major issue with some schools trying to get rid of computers 
because students have access at home and in public libraries. Computers, 
therefore, are seen by some as an urmecessary and huge expense for 
schools" (Province 2). This respondent explained that "because our 
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province doesn't have a mandated policy that schools must follow 
regarding technology, many schools are going on their own way." De- 
emphasizing technology was one of fhese ways. "Because of fhis 
variance across fhe province if will be a while before we in fesfing and 
evaluafion sfarf using some of fhese fechnologies in our assessmenfs" 
(Province 2). 

Concerns wifh fhe use of compufers and online fesfs in large-scale 
assessmenf also included issues of reliabilify and validity. One 
respondent explained that "one of fhe reasons why we're nof looking af 
fechnology or looking af fhe Nef in a more serious way is fhaf we see a 
lof of problems wifh reliabilify. Even securify, foo, becomes an issue" 
(Province 2). 

Tensions between political and educational uses of assessment results. 
Inferviewees in fhe Minisfries of Educafion saw fesf resulfs as mulfi- 
purpose. 

Our assessment tools are useful for piloting the system, for evaluating the 
system, for our institutional evaluation, that is to say for figuring out how our 
students fare against a specific standard. They are also used to illustrate 
coherence between practice and curriculum, and they are useful for the 
implementation of new curricula. (Province 3) 

Elowever, fhe people we inferviewed wresfled wifh fhe issue of 
using fhe resulfs of fheir large-scale assessmenfs for polifical and 
educafional purposes. They spoke of pressure from govemmenf fo be 
accounfable fo fhe public and how 

we in testing and evaluation have tried to make sure that these kinds of 
initiatives don't simply serve political ends, but they also serve our education 
systems, not so much as holding teachers and school boards accountable but 
more to identify students who are having difficulties so that we can support 
them. (Province 2) 

As more provinces move to publication of fhe large-scale assessmenf 
resulfs, one respondenf expressed growing concern abouf fhe public's 
abilify fo fully inferpref fhe resulfs. 
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People sometimes don't see all those factors that may influence the result. For 
example if we were to publish that our students in our schools were having 
difficulty using technology to find information on the web — that is a 
generalization that wouldn't be true in this school versus that school and could 
in fact be very damaging. (Province 2) 

CONCLUSION 

For this study, we were motivated by our desire to explore how current 
accountability measures in Canadian education address the new 
literacies of ICT, particularly those most relevant for new fimes. Through 
a series of pan-Canadian inferviews, we found fhaf Minisfry of 
Educafion personnel were wresfling wifh fhese issues and were af 
various sfages of irmovafion in fheir responses. 

Alfhough all provinces represenfed in our sfudy included a range of 
genres and use of visual fexfs in fheir reading and wrifing assessmenfs, 
fhey are, for fhe mosf parf, some years away from incorporafing ICT 
liferacies in any significanf way. The same frends were revealed in fhe 
U.S.A. where fhe majorify of sfafe liferacy fesfs do nof assess 
comprehension and evaluafion of information on fhe Infernef, and do 
nof allow use of word processors for assessing wrifing (Leu, Afaya, & 
Coiro 2002). As well, mosf of fhe individuals responsible for liferacy 
assessmenfs in bofh fhe U.S.A. and Canada regard locafing, evaluating, 
and comprehension of on-line maferials as a fechnology, nof a liferacy, 
issue. 

Plans for inclusion of fhe new liferacies of ICT varied among 
provinces from definife fo vague, wifh immediafe fo disfanf fimelines. 
Some aspecfs of ICT liferacies were more readily included fhan ofhers, 
and paper and pencil approaches, rafher fhan on-line confexfs, were 
more likely fo be used in provinces fhaf plan fo include ICT liferacies in 
fheir assessmenfs. 

Those responsible for large-scale assessmenfs grappled wifh issues 
fhaf direcfly influence fhe design of fhe insfrumenfs: concepfions of 
liferacy, relationships between the curriculum and assessment, access to 
and equitable opportunities to use ICT, and tensions between political 
and educational uses of assessmenf resulfs. There may be ofher interesf- 
based debafes relafed fo challenges facing fhose responsible for large- 
scale assessmenf, such as opposition from psychomefricians fo move 
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away from a print medium for testing, but these did not emerge in our 
data. 

TOWARDS AN EXTENDED AND BALANCED APPROACH TO 
LITERACY ASSESSMENT AND ACCOUNTABILITY 

Assuming that accountability in education is important and that 
assessment powerfully influences pedagogy, we found it encouraging 
that some provinces are begirming to develop multiple means of 
assessment. Ministry officials in two provinces planned in the next year 
to introduce portfolios on CDs at specific grade levels as part of their 
formalized assessments. However, the interviews reinforced how 
unreasonable it is to expect traditional assessment programs, as currently 
constructed, to adequately measure the skills required for twenty-first 
century living and working. A number of alternative assessment 
techniques could be included in students' portfolios such as performance 
assessment based on planning, undertaking, and completing a task or 
project, assessment of a larger unit of work, done either individually or 
as a group, which would facilitate developing the kinds of networking 
and social skills that learners will need increasingly in workplaces of the 
future. Innovation in technology directed towards knowledge 
mobilization could be developed in the service of giving multiple 
stakeholders access to such accounts of classroom work.^ 

However, moves in this direction will be successful only if, as the 
International Reading Association's (1999) position paper on assessment 
states, researchers "find ways to link performance assessment 
alternatives to questions that external audiences must address on a 
regular basis" (p. 7). These questions should include how Canadian 
schools are preparing students for the new literacies generally required 
for the new economy (Government of Canada, 2002). Models of 
alternative assessments that can answer this question are emerging in 
other educational systems such as in New Zealand and Great Britain 
(Johnston & Costello, 2005). If Canada moves to an assessment system 
that values teachers' informal assessments and on-going performance 
and portfolio assessments, as England has done, then teachers will need 
help in constructing more systematic and rigorous procedures that 
would include the reconceptualized notions of literacy discussed here. 
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Other considerations are a) the effects of bi-annual, even annual, 
assessmenfs - cycles fhaf increase pressure for compuferized responses, 
fhaf lead fo over sampling of narrow and/or fradifional aspecfs of 
liferacy and under-sampling of new liferacies and fhe insfrucfional 
implications fhaf follow; b) fhe use of alfemafive sampling sfrafegies so 
fhaf differenf children fake differenf secfions of an assessmenf as is done 
in New Zealand; and c) fhe use of aufhenfic performance fasks such as in 
New Zealand where children critically selecf from a sef of books fhose 
fhaf fhey deem should be in fheir library (Johnsfon & Cosfello, 2005). 
Finally, friangulafion of mulfiple dafa sources, rafher fhan single 
measures, is essential fo defermine a child's liferacy abilities fo give 
accounfs fo parenfs and fhe wider public. 

We do nof hold policy makers or fesf designers accounfable for fhe 
currenf dilemmas in and limifafions of large-scale, province-wide 
assessmenf. Our inferviewees were unfailingly committed fo fair and 
efhical pracfices and showed greaf concern for issues of equity. Rather, 
we suggest that the community of educators needs fo define fhe limifs of 
possibility for any form of single, large-scale assessmenf, fo bring rigour 
fo alfemafive forms, and fo raise awareness of reconcepfualized notions 
of liferacy and fhe affendanf assessmenf issues among parenfs and fhe 
public. If is in fhis spirif fhaf we undertook fhis sfudy. 

ACKNOWLEDGEMENT 

This study was funded by Social Sciences and Humanities Research Council 
of Canada Grants No. 410-2-2-0917 and No. 538-2002-1008. 

We want to acknowledge the generosity of the participants in giving so 
willingly of their time, and the contributions of the anonymous reviewers. 

NOTES 

' To preserve the anonymity of respondents, we identify provinces by 
numbers. Our intent was to provide a wholistic view of assessment in Canada, 
not to provide a comparison among provinces. We have edited quotations of 
those interviewed for smoother reading. 

^ The Multiliteracy Project (www.multiliteracies.ca) is currently developing 
such a system. 
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